Systems and methods for virtual reality motion sickness prevention

ABSTRACT

Systems and methods are disclosed herein for a sensory compensation device including a position and orientation sensor arranged to generate position and orientation data based on one or more of detected velocity, angular rate, gravity, motion, position and orientation associated with the device. The device also optionally includes an optical sensor arranged to capture real-time images and generate real-time image data of an area adjacent to the device. The device includes a processor arranged to: i) optionally receive the real-time image data, ii) receive the position and orientation data and iii) generate compensated image data based on the real-time image data and the position and orientation data. Furthermore, the device includes a display arranged to display compensated images derived from the compensated image data where a portion of the compensated images includes the captured real-time images, if captured, with adjusted positions and orientations in relation to the captured real-time images.

REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 62/519,287, filed on Jun. 14, 2017, and entitled“Virtual Reality Motion Sickness Prevention.” The entire contents of theabove-referenced application are incorporated herein by reference.

BACKGROUND

Many people are affected by motion sickness, whether resulting frombeing at sea (e.g., on a ship, submarine, diving, etc.), from flying,from driving, or even in other circumstances. Motion sickness can bedebilitating, resulting in nausea, vomiting and preventing people fromcarrying out their tasks as planned, whether it be continuing to carryout operations on a submarine, continuing to safely operate a plane, orcontinuing to enjoy a dive. As described in greater detail in U.S. Pat.No. 5,966,680 (the “'680 patent”), motion sickness results from amismatch between what motions a person sees happening, and what motionsthat person internally perceives. Motion sickness can result from amotion felt but not seen, a motion seen but not felt, or different typesor amounts of motions being felt and seen. For example, an individual ina ship sees indications of a certain motion or lack thereof in the ship(e.g., the person is in the same reference frame as the objects hesees), but that individual's vestibular system perceives a differentmotion (e.g., motion of the ship, yawing, pitching or rolling as awhole). Hence, there is a need for enabling a means to compensate forthe sensory mismatch leading to motion sickness.

One method for compensating a sensory mismatch, as described in the '680patent, is a wearable accessory providing visual orientation cues toreflect motion of the user with respect to the environment, in order totrain the user's brain to correct the sensory mismatch, i.e. to correctwhat the user perceives. In an alternative embodiment, the '680 patentdisplays no visual cues, but records an image and averages it with otherimages so as to produce what is perceived by the user to be a slowlychanging display of the visual environment, projected onto glasses,effectively minimizing the magnitude of any observed movement. Yet, sucha device and its visual cues require additional user training and userexperience to enable users to properly account for cues to alleviate theeffects caused by sensory mismatches.

Accordingly, there remains a need to provide persons subject to theadverse effects of sensory mismatch with systems, methods, and devicesthat more naturally and readily enable a user to overcome perceptionmismatches leading to such adverse effects.

SUMMARY

Systems and methods are disclosed herein for providing virtual realityand/or augmented reality mechanisms to alleviate or mitigate effectscaused by a person's sensory mismatch of their perceived surroundings.According to one aspect, a sensory compensation device includes aposition and orientation sensor arranged to generate orientation andacceleration data based on one or more of detected velocity, angularrate, gravity, motion, position, acceleration, and orientationassociated with the device. In some implementations, the device includesan optical sensor arranged to capture real-time images and generatereal-time image data of an area adjacent to the device. The devicefurther includes a processor arranged to: i) receive the real-time imagedata, ii) receive the orientation and acceleration data and iii)generate compensated image data based on the real-time image data andthe orientation and acceleration data. The device includes a displayarranged to display compensated images derived from the compensatedimage data such that a portion of the compensated images includes thecaptured real-time images with adjusted orientations and accelerationsin relation to the captured real-time images.

The orientation and acceleration data may include at least one of roll,pitch, and yaw data in relation to an inertial reference point. Theinertial reference point may include a reference horizon. Theorientation and acceleration data may include a differential positionbetween the inertial reference point and a detected orientation andacceleration of the device. The portion of compensated images may beadjusted proportional to the differential position between the inertialreference point and the detected position and orientation of the device.The portion of compensated images may be adjusted in a reciprocaldirection or orientation as the direction or orientation of the detecteddifferential position. The portion of compensated images may be rolledclockwise by one or more degrees proportional to a detected roll of thedevice by the one or more degrees in a counterclockwise direction.

The device may be head-mounted, hand-held, or detachably connectable toan optical system. The optical system or instrument may includeeyeglasses. The display may include LCD, LED, liquid crystal on silicon(LCoS), and/or OLED elements. The display may include a computermonitor, television, mobile device display, computer tablet display,projection from a projector onto a portion of a wall or bulkhead, and/orany other visual display apparatus.

The area adjacent to the device may include at least one of an areaforward, behind, and at a side of the device. The optical sensor mayinclude at least one video camera. The compensated image data mayinclude data that enables the display of at least one of an overlay,background, and shadow image with respect to the captured real-timeimages. At least one of the overlay, background, or shadow image mayinclude at least one of a reference horizon and sky. The sky may be anight sky including one or more stars.

In another aspect, a sensory compensation device includes an orientationand acceleration sensor arranged to generate orientation andacceleration data based on one or more of detected velocity, angularrate, gravity, motion, position, acceleration, and orientationassociated with the device. The device includes an optical elementthrough which a user views an area adjacent to the device. The devicealso includes a processor arranged to: i) receive the orientation andacceleration data and ii) generate orientation and acceleration imagedata. The device further includes a display arranged to displayorientation and acceleration images derived from the orientation andacceleration image data to the user via the optical element such thatthe orientation and acceleration images include one or more featureshaving adjusted orientations and accelerations in relation to the viewof the area adjacent to the device.

The position and orientation image data may include data that enablesthe display of at least one of an overlay image on the view of the areaadjacent to the device via the optical element. The overlay may includeat least one of a reference horizon and sky.

In another aspect, a computing device includes an orientation andacceleration sensor arranged to generate orientation and accelerationdata based on one or more of detected velocity, angular rate, gravity,motion, acceleration and orientation associated with the device. Thedevice also includes a processor arranged to: i) receive the orientationand acceleration data and ii) generate compensated image data based onthe orientation and acceleration data. The device further includes adisplay arranged to display compensated images derived from thecompensated image data, where a portion of the compensated imagesinclude one or more features having adjusted orientations andaccelerations in relation to a user's view of the area adjacent to thedevice. The device may be a computer desktop, laptop, mobile device,computer where the display is projected via a projector, or a portablecomputer tablet.

In a further aspect, a hand-held mobile computing device includes anorientation and acceleration sensor arranged to generate position andorientation data based on one or more of detected velocity, angularrate, gravity, motion, position, acceleration, and orientationassociated with the device. The device includes an optical sensorarranged to capture real-time images and generate real-time image dataof an area adjacent to the device. The device also includes a processorarranged to: i) receive the real-time image data, ii) receive theorientation and acceleration data and iii) generate compensated imagedata based on the real-time image data and the orientation andacceleration data. The device further includes a display arranged todisplay compensated images derived from the compensated image data suchthat a portion of the compensated images includes the captured real-timeimages with adjusted orientations and accelerations in relation to thecaptured real-time images.

Other objects, features, and advantages of the present invention willbecome apparent upon examining the following detailed description, takenin conjunction with the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The systems and methods described herein are set forth in the appendedclaims. However, for purpose of explanation, several illustrativeaspects are set forth in the following figures.

FIG. 1 depicts an optical device arranged to display an image to a useror augment a user's existing visual perception of their surroundings.

FIG. 2 is block diagram of an exemplary computer system for implementingat least a portion of the systems and methods described herein.

FIG. 3A is a conceptual diagram showing vertical and horizontal axesassociated with a vehicle in relation to a horizon.

FIG. 3B is a conceptual diagram illustrating pitch, roll, and yawassociated with a submarine.

FIG. 4A shows a display of an image provided by an optical device.

FIG. 4B illustrates a horizon.

FIG. 5A shows a display of an uncompensated image or direct visualperception when a roll by a degree alpha occurs

FIG. 5B illustrates a change in roll in relation to a horizon.

FIG. 6A shows a display of a compensated image when a counterclockwiseroll of alpha degrees is detected.

FIG. 6B illustrates the change in roll in relation to the horizon.

FIG. 7 is another display of an image including a horizon and night skywith at least one star or an overlay of a horizon and night sky with atleast one star over a user's perceived field of view.

FIG. 8 is a display of an image including a horizon and night sky withat least one star when a detected roll of alpha degrees is detected oran overly of a horizon and night sky with at least one star over auser's perceived field of view.

FIG. 9 depicts a virtual or augmented reality space surrounding a userincluding multiple virtual displays in various positions around theuser.

FIG. 10 depicts a virtual or augmented reality space surrounding a userincluding a display capable of being extended continuously around auser.

FIG. 11 is an exemplary process according to a method of alleviating theadverse effects of a sensory mismatch.

DETAILED DESCRIPTION

To provide an overall understanding of the invention, certainillustrative aspects will now be described. However, it will beunderstood by one or ordinary skill in the art that the systems andmethods described herein can be adapted and modified for other suitableapplications and that such other additions and modifications will notdepart from the scope hereof.

Individuals can be affected by motion sickness in moving vehicles suchas cars, trains, boats, ships, and airplanes. The effects of motionsickness can be severe and disabling. Motion sickness is often caused bya sensory mismatch between a person's visual perception of theirsurroundings and perceived motion via the person's inner ear (e.g.,vestibular system) and brain. In such circumstances, the person's braincannot reconcile one's sensed motion with what the person's eyes areseeing. Vertigo is one type of sensor mismatch where the body perceivesmotion when there is actually no motion, and attempts to counteract suchperceived motion by changing the body's posture.

FIG. 1 depicts an optical device 100 arranged to display an image to auser or augment a user's existing visual perception of theirsurroundings. The device 100 may include a housing 102 arranged to coverthe eyes of a user while providing a one or more display elements thatdisplay images and other information to the user. The device may includean optical sensor 104 such as a video camera arranged to capturereal-time video images in an area adjacent to the device 100 such as infront of, behind of, or next to the device 100. The device 100 mayinclude an orientation and acceleration sensor 106, e.g., anaccelerometer to detect velocity, angular rate, gravity, motion,position, acceleration, and/or orientation associated with device 100.The device 100 may include a computer system such as described withrespect to FIG. 2. In some implementations, the device 100 may include apair of eyeglasses. In such a configuration, the device 100 may notimplement optical sensor 104 because the user will be able to view thearea in front of the device 100 through the optical elements,eyeglasses, of the device 100. The device 100 may include a displayprojected onto a surface of the eyeglasses or directly to an eye or eyesof the user to present orientation and acceleration images to a user. Inone aspect, device 100 may include a computer, a laptop, or a tablet.

The orientation and acceleration sensor 106 may include at least oneaccelerometer. The accelerometer may detect the Earth's gravity based ondetecting an acceleration due to the Earth's gravity straight downwardsof about 9.81 m/s² at the Earth's surface. An accelerometer may be usedto determine the Earth's reference horizon because gravity always actsperpendicular to the Earth's surface. The reference horizon may bereferred to as the line at which the Earth's surface and the sky appearto meet. In some implementations, the accelerometer includes athree-access accelerometer. In some implementations, a sensor includingan accelerometer or other positional sensing element (or a processorreceiving sensor data) may establish an inertial reference point fromwhich subsequent displays to a user are adjusted based on sensed changesin position, acceleration, and/or orientation with respect to theinertial reference point. One component of the reference point mayinclude a plane corresponding to the Earth's horizon.

FIG. 2 is block diagram of an exemplary computer system 200 forimplementing at least a portion of the systems and methods describedherein, including implementation computer processing in device 100. Theexemplary system 200 includes a processor 202, a memory 208, and aninterconnect bus 218. The processor 202 may include a singlemicroprocessor or a plurality of microprocessors for configuringcomputer system 200 as a multi-processor system. The memory 208illustratively includes a main memory and a read-only memory. The system200 also includes the mass storage device 210 having, for example,various disk drives, tape drives, etc. The main memory 208 also includesdynamic random access memory (DRAM) and high-speed cache memory. Inoperation and use, the main memory 208 stores at least portions ofinstructions for execution by the processor 202 when processing data(e.g., model of the terrain) stored in main memory 208.

In some aspects, the system 200 may also include one or moreinput/output interfaces for communications, shown by way of example, asinterface 212 for data communications via the network 216. The datainterface 212 may be a modem, an Ethernet card or any other suitabledata communications device. The data interface 212 may provide arelatively high-speed link to a network 216, such as an intranet,internet, or the Internet, either directly or through another externalinterface. The communication link to the network 216 may be, forexample, any suitable link such as an optical, wired, or wireless (e.g.,via satellite or 802.11 Wi-Fi or cellular network) link. In someaspects, communications may occur over an acoustic modem. For instance,for AUVs, communications may occur over such a modem. Alternatively, thesystem 200 may include a host computer system capable of web-basedcommunications via the network 216. In some aspects, the system 200 alsoincludes suitable input/output ports or may use the Interconnect Bus 218for interconnection with a local display 204 and user interface 206(e.g., keyboard, mouse, touchscreen) or the like serving as a local userinterface for programming and/or data entry, retrieval, or manipulationpurposes. Alternatively, server operations personnel may interact withthe system 200 for controlling and/or programming the system from remoteterminal devices (not shown in the Figure) via the network 216.

In some aspects, a system requires a processor, such as in device 100,coupled to one or more sensors (e.g., optical sensor 104 and orientationand acceleration sensor 106). Data corresponding to sensed or detectedinformation by sensor 104 and/or sensor 106 may be stored in the memory208 or mass storage 210, and may be retrieved and/or received by theprocessor 202. Processor 202 may execute instructions stored in thesememory devices to perform any of the methods described in thisapplication, e.g., sensory compensation and/or generation of modified,adjusted and/or compensated displays of images to a user.

The system may include a display 204 for displaying information, amemory 208 (e.g., ROM, RAM, flash, etc.) for storing at least a portionof the aforementioned data, and a mass storage device 210 (e.g.,solid-state drive) for storing at least a portion of the aforementioneddata. Any set of the aforementioned components may be coupled to anetwork 216 via an input/output (I/O) interface 212. Each of theaforementioned components may communicate via interconnect bus 218.

In some aspects, the system requires a processor coupled to one or moresensors. The sensor 104 may include one or more image capture componentsand/or video cameras. The sensor 106 may include one or more inertialdetection elements such as, for example, one or more accelerometers orone or more gyroscopes.

Data corresponding to sensed video images and/or sensed orientation andacceleration, and/or changes in position, orientation, and acceleration,and a process for providing sensory compensation to a user or users maybe performed by a processor 202. The system may include a display 204for displaying information, a memory 208 (e.g., ROM, RAM, flash, etc.)for storing at least a portion of the aforementioned data, and a massstorage device 210 (e.g., solid-state drive) for storing at least aportion of the aforementioned data. Any set of the aforementionedcomponents may be coupled to a network 216 via an input/output (I/O)interface 212. Each of the aforementioned components may communicate viainterconnect bus 218. Orientation and acceleration data may include databased on detected velocity, angular rate, gravity, motion, position,acceleration, and orientation associated with the device.

In operation, a processor 202 receives data from the sensor(s) 104and/or 106, calculates a change in orientation and/or acceleration ofthe device 100 and user, and then modifies and/or adjusts aspects orportions of a images presented to the user as described herein to reducepossible adverse effects of sensory mismatches between the user's visualperception and inner ear. Optionally, the processor 202 may receiveadditional information from a vessel in which, for example, the device100 is located. The additional information may include, for example,navigational information, ship's status information, and/or altitudeinformation, and a processor 202 may include a portion of suchinformation and/or information as described in the '680 patent. Theoutput from the system processor 202 may directed to the display 204and/or transmitted to another display external to device 100, e.g., aship's control room display.

The components contained in the system 200 are those typically found ingeneral purpose computer systems used as servers, workstations, personalcomputers, network terminals, portable devices, mobile devices and thelike. In fact, these components are intended to represent a broadcategory of such computer components that are well known in the art.

It will be apparent to those of ordinary skill in the art that methodsinvolved in the systems and methods of the invention may be embodied ina computer program product that includes a non-transitory computerusable and/or readable medium. For example, such a computer usablemedium may consist of a read only memory device, such as a CD ROM disk,conventional ROM devices, or a random access memory, a hard drive deviceor a computer diskette, a flash memory, a DVD, or any like digitalmemory medium, having a computer readable program code stored thereon.

Optionally, the system 100 and/or 200 may include an inertial navigationsystem, a Doppler sensor, an altimeter, a gimbling system to fixate thesensor on a populated portion of a holographic map, a global positioningsystem (GPS), a long baseline (LBL) navigation system, an ultrashortbaseline (USBL) navigation, or any other suitable navigation system.

FIG. 3A is a two-dimensional cross-sectional diagram 300 showingvertical axis 304 (the “Z” axis) and a horizontal axis 302 (the “X”axis) associated with a vehicle 306 in relation to a reference horizon308. Although not shown, there is a “Y” axis extending through thecenter of vehicle 306 (i.e., a submarine's hull) from bow to sternintersecting the center point 312. For a vessel riding on the surface ofan ocean, the horizontal axis 302 should generally be parallel to theEarth's horizon or reference horizon 308. Also, the reference horizon308 should be perpendicular to the Earth's gravity 310, which has anacceleration relatively straight downward at the ocean surface.

FIG. 3B is a three-dimensional conceptual diagram 350 illustratingpitch, roll, and yaw associated with a submarine 352. Assuming that thesubmarine 352 is moving in the direction of the “Y” axis 354, roll 360represents the rotation or rotational orientation of the submarine alongits bow-to-stern axis 354. Pitch 362 represents the submarine's rotationor rotational orientation along its “X” axis 356 that extendsperpendicularly from a center point 366 of the submarine 352, but alsoin a substantially parallel direction as the horizon 308. Yaw 364represents the submarines rotation along its “Z” axis 358 which extendsin a substantially perpendicular direction to the reference horizon 308.A plane defined by the “X” axis 356 and “Y” axis 354 may also generallylie in parallel to the reference horizon 308 at least within thevicinity of the submarine 352. In some configurations, the orientationand acceleration sensor 106 senses, detects, and/or measures the roll360, pitch 362, and/or yaw 364 associated with device 100 while beinglocated within the submarine 352 as the submarine 352 moves through thewater and experiences changes in orientation and acceleration due to itssurrounding environment.

The device 100 may include an orientation and acceleration sensorarranged to generate orientation and acceleration data based on one ormore of detected velocity, angular rate, gravity, motion, position,acceleration, and orientation associated with the submarine 352 and,therefore, the device 100 because the device 100 is located in thesubmarine 352. However, the degree of changes in pitch 362 may varydepending on the location of the device 100 within the submarine 352.The sensor 106 may detect changes in roll 360, pitch 362, and/or yaw 364in relation to an inertial reference point such as a reference inrelation to a determined reference horizon 308, which may be included inthe orientation and acceleration data. For example, the determinedinertial reference point may correspond to the center point 366 ofsubmarine 352. The inertial reference point may be stationary orrelative. For example, submarines may use a stationary inertialreference point to enable underwater navigation over long or shortdistances. The submarine may periodically reset its inertial referencepoint by going to the surface and, for example, using GPS information tocalibrate or adjust its location information derived from inertialsensors. An inertial reference may be relative with respect to, forexample, the Earth's horizon or a reference horizon 308 which may bedetermined, for example, by detecting the Earth's gravitationalacceleration. An inertial reference point may be established by device100 when, for example, the device experiences less than a threshold ofacceleration. For example, when submarine 352 is stationary in calmwaters, the sensor 106 may only substantially detect the Earth'sgravitational acceleration.

The device 100 may use an optical sensor to capture real-time images andgenerate real-time image data of an area adjacent to the device. Thedevice 100 may include a processor arranged to: i) receive the real-timeimage data, ii) receive the orientation and acceleration data and iii)generate compensated image data based on the real-time image data andthe orientation and acceleration data. The device 100 may include adisplay, e.g., display 205, that displays compensated images derivedfrom the compensated image data such that a portion of the compensatedimages includes the captured real-time images with adjusted orientationsand accelerations in relation to the captured real-time images.

For a person on the submarine 352, the submariner's eyes are locked onwhat they see within the local reference frame of the submarine: avisual reference frame, e.g., a room, table, instruments, etc. When thesubmarine moves, i.e. yaws, pitches or rolls, what the person sees(their visual reference frame of the submarine) does not match what theyfeel: an inertial reference frame. In one implementation, a virtualreality system, e.g., device 100, displays to the user images adjustedto match the user's inertial reference frame.

FIG. 4A shows a reference horizon 401. FIG. 4B shows a view 403 seen bya user, including an object 407 positioned on a surface 405. Because thesubmarine is level, the reference horizon 401 is level, and the view 403is also level—there is no sensory mismatch.

FIG. 5A shows a reference horizon 501 (i.e., the Earth's horizon) whichis inclined relative to a horizontal axis 508, i.e., “X” axis 356 ofFIG. 3B, by an angle alpha, reflecting that the submarine has now rolledcounterclockwise by this angle alpha. As a result, because the user ispositioned on the submarine 352, FIG. 5B shows the view 503 seen by theuser still includes both the surface 505 and the object 507 positionedon that surface that appears level, even though the surface or deck isactually tilted (rolled) by alpha degrees in the counterclockwisedirection. The mismatch between the view 503 seen by the user, and theactual orientation of the submarine 352, reflected by the angle alpha ofthe reference horizon 501, is what leads to motion sickness.Additionally, the rate of change of the angle and/or acceleration ofchange can contribute to motion sickness The present invention is notlimited to roll by an angle alpha—any type of movement including anycombinations of yaw, pitch, roll, along with any translational motionmay be adjusted for, as further described.

FIGS. 6A and 6B illustrate how a compensated image may be used toalleviate the adverse effects of sensory mismatches according to thepresent invention. In this implementation, FIG. 6B shows the referencehorizon 601 is still inclined by an angle alpha, indicating again thatthe submarine rolled counterclockwise by this angle alpha. The systemand/or device 100 records video of what optical sensor 104 captures,e.g., what the user would see in FIG. 5 when the submarine rollscounterclockwise. The system further detects through one or more sensors106 (e.g., accelerometers) the counterclockwise roll of the ship by anangle alpha. The system further processes the recorded video beforedisplaying the processed video in real-time or near-real time to theuser, to match the detected movement, as shown in the view 603 that theuser is now seeing. The processed video (e.g., the recorded videorotated by the same angle alpha, but in the clockwise direction), alsoreferred to as compensated images, is generated for display in real-timeor near-real time to the user, such that the user sees what is shown inFIG. 6A. Because what the user sees (e.g., a roll by an angle alpha)reflects what the user feels (e.g., a roll of the submarine by thatangle alpha), motion sickness can be avoided. In other words, the tiltor roll by alpha degrees on objects 605 and 607 gives the visualsensation to the user of tilt or roll in the counterclockwise directionby alpha degrees, which corresponds to the user's sensation of acounterclockwise roll. One technical advantage of this implementationincludes, for example, a capability for a sailor to more efficientlyreview documents on a ship in turbulent waters. The virtual realitysystem, worn by the sailor, detects the ship's movements in theturbulent waters, and adjusts the display of the documents to match thesubmarine movements. Because what the sailor sees (the processed displayof the documents), matches what is perceived by the sailor's vestibularsystem (movement of the ship in turbulent waters), including changes inangles at certain rates and/or accelerations, motion sickness and itsside effects can be avoided.

Processing of the recorded video may include parsing certain portions ofthe recorded video and rotating, scaling, or otherwise adjusting one ormore portions of the recorded video. Similarly, processing of therecorded video may also include parsing certain sequences of therecorded video, and rotating, scaling, or otherwise adjusting one ormore time segments of the recorded video. In some embodiments, theprocessed video may be displayed to the user through a virtual realitydevice, such as the Microsoft Oculus VR headset, an augmented realitydevice or any other virtual reality headset (e.g., device 100). In someembodiments, the processed video may be generated for display throughgoggles compatible with any handheld mobile device, such as a phone ortablet. In some embodiments, the processed video may be generated fordisplay as a 3D environment. In some embodiments, the processed videomay be generated for display in a spherical display, with differenttiles showing different views, the system automatically adjusting theposition of the titles as a function of the detected user's movement,such that the user is shown the one or more tiles best corresponding tothe movement that user is experiencing. In some embodiments the tilesmay show the same view so that as one tile rotates out of the field ofview the user can refocus on a tile moving into the field of view andcontinue their activity. In some embodiments the tiles may show acomputer desktop with applications. In some embodiments the tiles mayshow a Unix terminal, an Emacs window, or a Microsoft Word document. Insome embodiments the tiles may show an integrated developmentenvironment such as an Eclipse that allows the user to program incomputer languages such as C++ or Java.

FIG. 7 is another display of an image 700 including a reference horizon704 and night sky 702 with at least one star 706 or an overlay on auser's direct view through an optical instrument (e.g., eyeglasses)including a reference horizon 704 and night sky 702 with at least onestar 706. In one configuration, device 100, operating as a VR headset,displays image 700. In a submarine or an enclosed space in a ship orplane, a user is unable to perceive the Earth's horizon. In FIG. 7, thedevice 100 displays a view of the area in front of the device 100 basedon video images captured by optical sensor 104. The originally capturedvideo images and data from sensor 104 have been modified to include areference horizon 704 and/or sky 702. The horizon 704 may be displayedas an overlay, shadow, or background in relation to the originallycaptured video images from sensor 104. The reference horizon 704 and/orsky 702 enable a viewer to visually perceive changes in the orientationand acceleration of the submarine (or other vessel) that correlate withthe viewer sensation of changes by their inner ear to alleviate motionsickness.

FIG. 8 is a display of an image 800 including a reference horizon 804and night sky 802 with at least one star 806 when a roll of alphadegrees is detected. The reference horizon 804 and/or sky 802 enable aviewer to visually perceive the roll of the submarine (or other vessel)by alpha degrees by showing a tilt or roll of the horizon 804 by alphadegrees. This enable viewer to correlate their visual perception withtheir motion sensations by their inner ear to alleviate motion sickness.Again, the originally captured video images and data from sensor 104have been modified to include a reference horizon 804 and/or sky 802that are tilted by alpha degrees. The horizon 804 may be displayed as anoverlay, shadow, or background in relation to the originally capturedvideo images from sensor 104.

FIG. 8 also illustrates overlay on a user's direct view through anoptical instrument (e.g., eyeglasses) including a reference horizon 804and night sky 802 with at least one star 806. In this implementation,the device 100 is configured to allow the viewer to directly view anarea adjacent to them (i.e., in front of them), but the device 100provided an overlay with respect to at least one eye of the view thatdisplays the horizon 804, sky 802, and/or star 806. In a furtherimplementation, the device 100 is configured as an attachablydisconnectable unit capable of physical attachment to eyeglasses wherebyan overlay of the horizon 804, sky, and/or start 806 are displayed to atleast one eye of a viewer using the eyeglasses. Instead of modifying theposition, acceleration, and orientation of objects as described withrespect to FIGS. 6A and 6B, features are added to the captured image oroverlaid onto a direct view via an optical instrument to enable a viewerto perceive the orientation and acceleration of the reference horizonwhich should always correlate with the viewer's perceived vestibularorientation and acceleration.

The change in orientation and acceleration displayed detected may bedisplayed proportionally by device 100. For example, a detected changein pitch of beta degrees may be displayed as a change in pitch by betadegrees. Optionally, the display change may be a fraction or portion ofthe detected change in pitch, roll, and/or yaw. Although FIGS. 6B and 8are examples of how the device 100 processes rolls, device 100 appliessimilar techniques with respect to detected changes in pitch and yaw. Incertain implementations, device 100 concurrently detects and processeschanges in roll, pitch, and yaw, and generates compensated images basedon the combination of changes associated with roll, pitch, and yaw.

FIG. 9 depicts a virtual or augmented reality space 900 sphericallysurrounding a viewer 902 of device 100 including multiple virtualdisplays or tiles 904, 906, 908, 910, 912, and 914 in various positionsaround the viewer. In some implementations, the processed video orcompensated images may be generated for display in a spherical display,with different tiles showing different views, the system mayautomatically adjust the position of one or more of the titles as afunction of the device's or viewer's movement, such that the viewer isshown the one or more tiles best corresponding to the movement thatviewer is experiencing. The position, acceleration, and orientation ofeach tile may be modified in response to detected changes in roll,pitch, and yaw. For example, if device 100 detects a pitch downward ofbeta degrees, the tile 904 may be shifted upward by beta degreecorrespondingly to give the visual impression of movement upward to user902 which will correspond to the user's vestibular sensation of adownward pitch. In some implementations, the image a tile 904, 906, 908,910, 912, and/or 914 may be modified, adjusted, or compensated in amanner described with respect to FIG. 6B or 8.

FIG. 10 depicts a virtual or augmented reality space 1000 surrounding aviewer 1002 including a display 1004 capable of being virtually extendedcontinuously around the viewer 1002. Instead of presenting multipletiles, device 100 may virtually stitch together multiple tiles such astiles 904-914 to provide a more seemly transition among the differenttiles. Portions of the display 1004 may shift in position, acceleration,and orientation in a similar manner as the tiles in FIG. 9. Thetechniques described with respect to FIGS. 6A-10 may be applied using ahand-held mobile device using an augmented reality application. Forinstance, a mobile device with a camera may be held by a viewer thatviews the touch screen. The camera may capture real-time video anddisplay the captured video on the display, while the processor, runninga sensory compensation application, overlays a reference horizon and/orsky onto the displayed image.

FIG. 11 is an exemplary process 1100 for alleviating the adverse effectsof a sensory mismatch. First, generate orientation and acceleration databased on one or more of detected velocity, angular rate, gravity,motion, position, acceleration, and orientation associated with thedevice 100 using a sensor 106 (Step 1102). Capture real-time imagesusing an optical sensor 104 and generate real-time image data of an areaadjacent to the device (Step 1104). Using a processor, receive thereal-time image data, receive the orientation and acceleration data andgenerate compensated image data based on the real-time image data andthe orientation and acceleration data (Step 1106). Then, displaycompensated images derived from the compensated image data such that aportion of the compensated images includes the captured real-time imageswith adjusted orientations and accelerations in relation to the capturedreal-time images (Step 1108).

As previously mentioned, the present invention applies equally to anytype motion sickness beyond seasickness, such as carsickness orairsickness. In one example, a young child is riding in a car whilereading a book. As the car turns a corner, the young child mayexperience a sensory mismatch. The virtual reality system, worn by theyoung child, detects the acceleration and any yaw, roll or pitchresulting from the car turning a corner, and adjusts the displayaccordingly. Similarly, a pilot is making a turn with his plane. Insteadof displaying on a dial a visual of the airplane's yaw, roll or pitch,the visual reality system acquires information regarding the yaw, rollor pitch, captures data (e.g., what the pilot would normally see in hiscockpit), processes said data to account for the yaw, roll or pitch, anddisplays the processed data for the pilot to see.

Various features, methods, systems, and devices described herein may becombined with features described in U.S. Pat. No. 5,966,680, the entirecontents of which are incorporated herein by reference.

It will be apparent to those skilled in the art that such aspects areprovided by way of example only. It should be understood that numerousvariations, alternatives, changes, and substitutions may be employed bythose skilled in the art in practicing the invention.

Accordingly, it will be understood that the invention is not to belimited to the aspects disclosed herein, but is to be understood fromthe following claims, which are to be interpreted as broadly as allowedunder the law.

What is claimed is:
 1. A sensory compensation device comprising: anorientation and acceleration sensor arranged to generate orientation andacceleration data based on one or more of detected velocity, angularrate, gravity, motion, position, acceleration, and orientationassociated with the sensory compensation device; wherein the orientationand acceleration data include: a differential position between aninertial reference point and a detected orientation and acceleration ofthe device, and at least one of roll, pitch, and yaw data in relation tothe inertial reference point; an optical sensor arranged to capturereal-time images and generate real-time image data of an area adjacentto the sensory compensation device: a processor arranged to: i) receivethe real-time image data, ii) receive the position and orientation dataand iii) generate compensated image data based on the real-time imagedata and the orientation and acceleration data; and a display arrangedto display compensated images derived from the compensated image data,wherein a portion of the compensated images includes the capturedreal-time images with adjusted orientations and accelerations inrelation to the captured real-time images, and wherein the portion ofcompensated images is adjusted proportional to the differential positionbetween the inertial reference point and the detected orientation andacceleration of the device and in a reciprocal direction as thedirection of the change in the detected differential position.
 2. Thesensory compensation device of claim 1, wherein the inertial referencepoint includes a reference horizon.
 3. The sensory compensation deviceof claim 1, wherein the portion of compensated images is rolledclockwise by one or more degrees proportional to a detected roll of thesensory compensation device by the one or more degrees in acounterclockwise direction.
 4. The sensory compensation device of claim1, wherein the sensory compensation device is one of head-mounted,hand-held, and detachably connectable to an optical system.
 5. Thesensory compensation device of claim 4, wherein the optical systemincludes eyeglasses.
 6. The sensory compensation device of claim 1,wherein the display includes at least one of LCD, LED, liquid crystal onsilicon (LCoS), and OLED elements.
 7. The sensory compensation device ofclaim 1, wherein the area adjacent to the sensory compensation deviceincludes at least one of an area forward, behind, and at a side of thedevice.
 8. The sensory compensation device of claim 1, wherein theoptical sensor includes at least one video camera.
 9. The sensorycompensation device of claim 1, wherein the compensated image dataincludes data that enables the display of at least one of an overlay,background, and shadow image with respect to the captured real-timeimages.
 10. The sensory compensation device of claim 9, wherein the atleast one of the overlay, background, or shadow image includes at leastone of a reference horizon and sky.
 11. The sensory compensation deviceof claim 10, wherein the sky is a night sky including one or more stars.12. The sensory compensation device of claim 1, wherein the compensatedimages include navigational information.
 13. A sensory compensationdevice comprising: an orientation and acceleration sensor arranged togenerate orientation and acceleration data based on one or more ofdetected velocity, angular rate, gravity, motion, position andorientation associated with the sensory compensation device; wherein theorientation and acceleration data include: a differential positionbetween an inertial reference point and a detected orientation andacceleration of the device, and at least one of roll, pitch, and yawdata in relation to the inertial reference point: an optical elementthrough which a user views an area adjacent to the sensory compensationdevice; a processor arranged to: i) receive the orientation andacceleration data and ii) generate orientation and acceleration imagedata; and a display arranged to display orientation and accelerationimages derived from the orientation and acceleration image data to theuser via the optical element, wherein the orientation and accelerationimages include one or more features having adjusted orientations andaccelerations in relation to the view of the area adjacent to thesensory compensation device, and wherein the orientation andacceleration images are adjusted proportional to the differentialposition between the inertial reference point and the detectedorientation and acceleration of the device and in a reciprocal directionas the direction of the change in the detected differential position.14. The sensory compensation device of claim 13, wherein the orientationand acceleration image data includes data that enables the display of atleast one of an overlay image on the view of the area adjacent to thedevice via the optical element.
 15. The sensory compensation device ofclaim 14, wherein the overlay includes at least one of a referencehorizon and sky.
 16. A computing device comprising: an orientation andacceleration sensor arranged to generate orientation and accelerationdata based on one or more of detected velocity, angular rate, gravity,motion, acceleration and orientation associated with the computingdevice; wherein the orientation and acceleration data include: adifferential position between an inertial reference point and a detectedorientation and acceleration of the device, and at least one of roll,pitch, and yaw data in relation to the inertial reference point; aprocessor arranged to: i) receive the orientation and acceleration dataand ii) generate compensated image data based on the orientation andacceleration data; and a display arranged to display compensated imagesderived from the compensated image data, wherein a portion of thecompensated images include one or more features having adjustedorientations and accelerations in relation to a user's view of the areaadjacent to the computing device, and wherein the portion of compensatedimages is adjusted proportional to the differential position between theinertial reference point and the detected orientation and accelerationof the device and in a reciprocal direction as the direction of thechange in the detected differential position.
 17. The computing deviceof claim 16, wherein the computing device is one or more of a computerdesktop, mobile device, computer where the display is projected via aprojector, and portable computer tablet.
 18. A hand-held mobilecomputing device comprising: an orientation and acceleration sensorarranged to generate orientation and acceleration data based on one ormore of detected velocity, angular rate, gravity, motion, position,acceleration, and orientation associated with the hand-held mobilecomputing device; wherein the orientation and acceleration data include:a differential position between an inertial reference point and adetected orientation and acceleration of the device, and at least one ofroll, pitch, and yaw data in relation to the inertial reference point;an optical sensor arranged to capture real-time images and generatereal-time image data of an area adjacent to the hand-held mobilecomputing device; a processor arranged to: i) receive the real-timeimage data, ii) receive the orientation and acceleration data and iii)generate compensated image data based on the real-time image data andthe position and orientation data; and a display arranged to displaycompensated images derived from the compensated image data, wherein aportion of the compensated images includes the captured real-time imageswith adjusted orientations and accelerations in relation to the capturedreal-time images, and wherein the portion of compensated images isadjusted proportional to the differential position between the inertialreference point and the detected orientation and acceleration of thedevice and in a reciprocal direction as the direction of the change inthe detected differential position.